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Background / Context: 

Cluster randomized trials (CRTs), or studies in which intact groups of individuals are randomly 
assigned to a condition, are becoming more common in evaluation studies of educational 
programs. A specific type of CRT in which clusters are randomly assigned to treatment within 
blocks or sites, kn own as multisite cluster randomized trials (MSCRTs), are the most frequent in 
the literature. In a recent review, Spybrook, Shi, & Kelcey (under review) noted that of 22 CRTs 
funded by the Institute of Education Sciences between 2011 and 2013, almost half (n=12) of the 
studies were MSCRTs in which students were nested in clusters and clusters were randomly 
assigned to condition within blocks or sites. 

The primary question that often guides the design of MSCRTs is whether or not the program 
works. Hence the MSCRT is designed with the goal of being powered to detect the main effect 
of treatment. However, there are other important questions that one should consider during the 
design phase of the study. For example, for whom is the treatment effective? In what types of 
schools is the treatment effective? Hence the power to detect moderator effects may often enter 
the design considerations. 

The power to detect moderator effects for MSCRTs at the student, cluster, or site level has a 
much smaller literature base than the power to detect the main effect of treatment for MSCRTs. 
In 2000, Raudenbush and Liu examined power calculations for multi-site studies in which 
individuals are randomly assigned within sites. Spybrook (2014) provided empirical estimates of 
minimum detectable effect sizes for moderator effects in a sample of MSCRTs funded by IES. 
Bloom & Spybrook (2013) calculated power to detect moderator effects as well as variability in 
treatment effects across sites for MSCRTs. 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this paper is to extend power calculations to moderator effects in MSCRTs. 
Specifically we consider 3 -level MSCRTs in which individuals are nested within clusters and 
clusters are randomized within sites, and 4-level MSCRTs in which individuals are nested within 
sub-clusters, sub-clusters are nested within clusters and clusters are randomized within sites. We 
consider moderators at the individual, cluster, and site level and provide R code for the 
calculations. We demonstrate how power for main effects and moderator effects might both be 
considered in the design of a MSCRT. 


Significance / Novelty of study: 

This paper extends the power calculations for MSCRTs beyond the main effect of treatment to 
allow researchers to also consider the power for moderator effects in the design phase of the 
study. Currently, there is no user-friendly software for calculating power for moderator effects in 
MSCRTs. The R code provided in this study will help fill this gap in the resources available to 
researchers planning MSCRTs. 
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Statistical, Measurement, or Econometric Model: 

We provide the models for a 3-level MSCRT. For purposes of the proposal, we focus on the 
main effect of treatment and moderator effects at the individual, cluster, and site level. The full 
paper includes all the models and the power calculations for the 4-level MSCRT. 

Main effect of Treatment 

The main effect of treatment is often the primary interest in a MSCRT. In this case, suppose we 
have students (level- 1) nested within schools (level-2) and schools are randomly assigned to 
condition within districts (level-3). In the case of no moderator, the student level model is: 

Y ijk = x 0 jk + e ijk e ijk ~ N(0,<j 2 ) [ 1 ] 

for i e {1,2,..., «} persons per cluster, j e (1,2,..., J} clusters and k e {1,2,... , K) sites, 
where n 0jk is the mean for cluster j in site t, e jk is the error associated with each student; 

and cr 2 is the within-cluster variance. 

The level-2 model, or cluster-level model, is: 

jk = Am + Po\ k Wjk + r 0J k r ojk ~ ^( 0 , T K ) [ 2 ] 

where fi mk is the mean for site k; J3 0lk is the treatment effect at site h, W jk is a treatment 
contrast indicator, Vi for treatment and -Vi for the control ; r 0jk is the random effect associated 
with each cluster; andr is the variance between clusters within sites. 

The level-3 model, or site-level model, is: 

Pm — Vooo + U 00k var ( u 00k ) ~ T /3 m 

Poik ~ Toio + u o\ k var (u 01k ) ~ t ^ cov(w 00i _,w 01i _) = t p m [3] 

where y 000 is the grand mean; ;/ ol(l is the average treatment effect (“main effect of treatment”); 
u 00k is the random effect associated with each site mean; u m is the random effect associated 
with each site treatment effect; r p is the variance between site means; r /; is the variance 
between sites on the treatment effect; and z p is the covariance between site-specific means and 

site-specific treatment effects. Note that we allow the treatment effect to vary randomly across 
sites, however, we could also treat this as a fixed effect. 


The estimate of the treatment effect and the variance of the estimated treatment effect are: 

foio -Ye-Yc 
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[4] 


r 0 ioJ = [ r A 1 +4 ( r x+v 2 !") u]lK 

The power for the test for the main effect of treatment for the 3-level MSCRT, H 0 : / 0]0 = 0 , 

follows the same logic as the power for the main effect of treatment for the 2 -level CRT 
(Raudenbush, 1997). The /'statistic in this case though is a ratio MStreatment to the 
MStreatmentbyciuster. The ratio of expected mean squares is equivalent to 1+ A, where the 
noncentrality parameter is defined as: 


A = 


r A, + 4 l 


rl io 


zy + cr / n 


)lj\lK 


or 


A = 


a l + 4 (p + (1 -p)/n)/J IK 


[5] 


where 


(7 = 


/oio 




>p = - 


+ cr 


zr + cr" 


.2 r fl i 
d ZT + cr 2 


We standardize the parameters by the sum of the within site variance. As the noncentrality 
parameter increases, the power of the test increases. Note that the model above could easily be 
extended to include covariates to increase the precision of the estimate. 

Moderator Effects 

Given the space limitations in the proposal, we do not provide the models for each of the 
possible moderator effects: students, school, and district level. Instead, we provide the non- 
centrality parameter for each case. The parameters are standardized as shown in equation (5) for 
all cases. In each case, we assume a binary moderator, i.e. gender, 75 percent or greater 
free/reduced lunch in school vs. less than 75 percent free/reduced lunch in school. For student 
and school moderators, we treat them as fixed effects in higher level of the model. Note that we 
assume one moderator at the specific level in each case. Table 1 provides the noncentrality 
parameter for each of the three cases. 

Insert Table 1 about here 


Findings / Results: 

Table 1 suggests that the influence of the sample sizes at each level is not the same across the 
moderator effects and the main effect of treatment. That is, for the individual level moderator 
effect, the number of individuals per cluster has more influence in terms of increasing power 
than it does for the main effect or cluster or site moderator effect. The power to detect the cluster 
level moderator effect is more greatly influenced by the number of individuals per cluster and 
number of clusters per site than the power for the main effect or site moderator effect. Finally, 
the power for the site level moderator effect is influenced primarily by the number of sites, 
similar to the main effect, however the within site variance is four times that of the main effect of 
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treatment, suggesting larger numbers of sites are needed to detect site moderator effects than the 
main effect. 


Conclusions: 

It is critical to consider not only the main effect of treatment in the design of MSCRTs but also 
important moderator effects such as whether the treatment if more effective for boys than girls, 
or in schools with a high percent of kids with free/reduced lunch versus schools with a low 
percent of kids with free/reduced lunch. Three sample sizes contribute to the power in all cases, 
the number of individuals per cluster, the number of clusters per site, and the total number of 
sites. The relative influence of each sample size varies depending on the effect of interest: main 
effect, individual level moderator effect, cluster level moderator effect, or site level moderator 
effect. For example, increasing the number of individuals per cluster yields greater increases in 
power for the individual level moderator than in would for the main effect of treatment. Similarly 
increasing the number of sites is critical if the goal is to power for the main effect of treatment 
and the site level moderator. The important effects of interest, i.e. main effects and specific 
moderator effect, should be considered in the design phase of the study in order to try to 
maximize the likelihood that a study is powered to answer the primary questions. 
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Appendix B. Tables and Figures 

Table 1. The standardized noncentrality parameter for each type of moderator effect. 


Individual Level 
Moderator* 

Cluster Level 
Moderator** 

Site Level 
Moderator*** 

f- KS * 

l- K * 

CN 

1 

co + 4 (k + 4(1 -p)/n)t J 

v + \6{p + {\- p) ! n )/ J 

4cT|j +16(/? + (l -p)/n)/J 


*For individual level moderator, <x> is standardized variance of moderator effect across sites, 
k is standardized variance of moderator within sites, 8 is the standardized individual 
moderator effect, all other parameters are same as in case of main effect of treatment. 

**For cluster level moderator, v is the standardized variance of the moderator effect across 
sites, 8 is the standardized individual moderator effect, all other parameters are same as in 
case of main effect of treatment. 

***For site level moderator, cr j v i s the standardized residual variance of the moderator effect 

across, 8 is the standardized individual moderator effect, all other parameters are same as in 
case of main effect of treatment. 
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